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Abstract. We formulate and study a fundamental search and detection 
problem, Schedule Optimization, motivated by a variety of real-world 
applications, ranging from monitoring content changes on the web, social 
networks, and user activities to detecting failure on large systems with 
many individual machines. 

We consider a large system consists of many nodes, where each node 
has its own rate of generating new events, or items. A monitoring ap¬ 
plication can probe a small number of nodes at each step, and our goal 
is to compute a probing schedule that minimizes the expected number 
of undiscovered items at the system, or equivalently, minimizes the ex¬ 
pected time to discover a new item in the system. 

We study the Schedule Optimization problem both for deterministic and 
randomized memory less algorithms. We provide lower bounds on the 
cost of an optimal schedule and construct close to optimal schedules 
with rigorous mathematical guarantees. Finally, we present an adaptive 
algorithm that starts with no prior information on the system and con¬ 
verges to the optimal memoryless algorithms by adapting to observed 
data. 


1 Introduction 

We introduce and study a fundamental stochastic search and detection problem. 
Schedule Optimization, that captures a variety of practical applications, ranging 
from monitoring content changes on the web, social networks, and user activities 
to detecting failure on large systems with many individual machines. 

Our optimization problem consists of a large set of units, or nodes, that 
generate events, or items, according to a random process with known or unknown 
parameters. A detection algorithm can discover new items in the system by 
probing a small number of nodes in each step. This setting defines a discrete, 
infinite time process, and the goal of the stochastic optimization problem is to 
construct a probing schedule that minimizes the long term expected number of 
undiscovered items in the system, or equivalently, minimizes the expected time 
to discover a new item in the system. 

We outline several important applications of this schedule optimization prob¬ 
lem: 


News and Feed Aggregators. To provide up to date summary of the news, news 
aggregator sites need to constantly browse the Web, and often also the blogo- 
sphere and social networks, for new items. Scanning a site for new items requires 
significant communication and computation resources, thus the news aggregator 
can scan only a few sites simultaneously. The frequency of visiting a site has to 
depend on the likelihood of finding new items in that site. mm 

Algorithmic Trading on Data. An emerging trend in algorithmic stock trading 
is the use of automatic search through the Web, the blogosphere, and social 
networks for relevant information that can be used in fast trading, before it ap¬ 
pears in the more popular news sites |4ll4IKI17llll8rTO| . The critical issue in this 
application is the speed of discovering new events, but again there is a resource 
limit on the number of sites that the search algorithm can scan simultaneously. 

Detecting Anomaly and Machine Malfunction. In large server farm or any other 
large collection of semi-autonomous machines a central controller needs to iden¬ 
tify and contain anomalies and malefactions as soon as possible, before they 
spread in the system. To minimize interference with the system’s operation the 
controller must probe only a small number of machines in each step. 


1.1 Our Contribution 

We consider an infinite, discrete time process in which n nodes generate new 
items according to a stochastic process which is governed by a generating vector 
TT (see Section 13] for details). An algorithm can probe up to c nodes per step to 
discover all new items in these nodes. The goal is to minimize the cost of the 
algorithm (or the probing schedule), which we define as the long term (steady 
state) expected number of undiscovered items in the system. 

We first show that the obvious approach of probing at each step the nodes 
with maximum expected number of undiscovered items at that step is not opti¬ 
mal. In fact, the cost of such a schedule can be arbitrary far from the optimal. 

Our first result toward the study of efficient schedules is a lower bound on 
the cost of any deterministic or random schedule as a function of the generating 
vector TT. 

Next we assume that the generating vector tt is known and study explicit 
constructions of deterministic and random schedules. We construct a determin¬ 
istic schedule whose cost is within a factor of (3 -I- (c — l)/c) of the optimal cost, 
and a very simple, memoryless random schedule with cost that is within a factor 
of (2 -|- (c — l)/c) from optimal, where c is the maximum number of probes at 
each step. 

Finally, we address the more realistic scenario in which the generating vector, 
TT, is not known to the algorithm and may change in time. We construct an 
adaptive scheduling algorithm that learns from probing the nodes and converges 
to the optimal memoryless random schedule. 






2 Related Work 


The News and Feed Aggregation problem is a very well-studied topic, in which 
the general goal is to obtain the updates of news websites (e.g. by RSS feeds). 
Among many introduced objectives mm in studying this problem, the most 
similar one to our cost function is the delay function presented by [21] . In [21] it 
is assumed that the rates of the news publication does not change, where in our 
setting these rates may change and our algorithm (Adaptive) can adapt itself 
to the new setting. Also, we assume at any given time the number of probes is 
fixed (or bounded) regarding the limited computational power for simultaneous 
probes, but m uses a relaxed assumption by fixing the number of probes over 
a time window of a fixed length which may result in high number of probes at a 
single time step. Finally, |21) introduces a deterministic algorithm in which the 
number of probes to each feed is obtained by applying the Lagrange multipliers 
method (very similar result to Theorem |31), but they loose the guarantee on 
optimality of their solution, by rounding the estimated number of probes to 
integers. In contrast, our solution provides theoretical guarantee on optimality 
of our output schedule. 

Web-crawling is another related topic, where a web-crawler aims to obtain 
the most recent snapshots of the web. However, it differs from our model substan¬ 
tially: in web-crawling algorithm data get updated, so missing some intermediate 
snapshot would not affect the quality of the algorithm, where in our model data 
are generated and they all need to be processed [3122]. 

There has been an extensive work on Outbreak Detection (motivated in part 
by the “Battle of Water Sensors Network” challenge [20]) using statistic or mobile 
sensor in physical domains, and regarding a variety of objectives PITOlTj . Our 
model deviates from the Outbreak Detection problem as it is geared to detection 
in virtual networks such as the Web or social networks embedded in the Internet, 
where a monitor can reach (almost) any node at about the same cost. 

Another related problem is the Emerging Topie Detection problem, where the 
goal is to identify emergent topics in a social network, assuming full access to 
the stream of all postings. Besides having different objectives, our model differs 
mainly in this accessibility assumption: the social network providers have an 
immediate access to all tweets or postings as they are submitted to their servers, 
whereas in our model we consider an outside observer who needs an efficient 
mechanism to monitor changes, without having such full access privilege mM- 

In the next section, we formally define our model and the Schedule Opti¬ 
mization problem. 


3 Model and Problem Definition 

We study an infinite, discrete time process in which a set of n nodes, indexed 
by 1,... ,n, generate new items according to a random generating process. The 
generating process at a given time step is characterized by a generating vector 
TT = (tti, ... ,7r„), where is the expected number of new items generated at 










node i at that step (by either a Bernoulli or a Poisson process). The generation 
processes in different nodes are independent. 

We focus first on a static generating process in which the generating vector 
does not change in time. We then extend our results to adapt to generating 
vectors that change in time. 

Our goal is to detect new events as fast as possible by probing in each step 
a small number of nodes. In particular, we consider probing schedules that can 
probe up to c nodes per step. 

Definition 1 (Schedule). A c-schedule is a function 5 : N — >■ n}° 

specifying a set of c nodes to he probed at any time t € N. d deterministic 
function S defines a deterministic schedule, otherwise the schedule is random. 

Definition 2 (Memoryless Schedule). A random schedule is memoryless if 
it is defined by a vector p = (pi,... ,p„) such that at any step the schedule probes 
a set C of c items with probability JIjgcP* independent of any other event. In 
that case we use the notation S = p. 

Definition 3 (Cyclic Schedule). A schedule, S, is Ocyclic if there is a finite 
time to such that from time to on, the schedule repeats itself every period of i 
steps. A schedule is cyclic if it is i-cyclic for some positive integer I. 

The quality of a probing schedule is measured by the speed in which it 
discovers new items in the system. When a schedule probes a node i at a time 
t, all items that were generated at that node by time t — 1 are discovered (thus, 
each item is not discovered in at least one step). We define the cost of a probing 
schedule as the long term expected number of undiscovered items in the system. 

Definition 4 (Cost). The cost of schedule S in a system of n nodes with gen¬ 
erating vector TT is 


1 


t 


eostiS,n) = ^Ihn ^ ^ E [Q^{T)] = ^lim - ^ ^ 

t' — l t' — l i—1 


E [Qfit')] , 


where Qf{t') is the number of undiscovered items at node i and at time t', and 
Q^{t') = '^^^iQf{t'). The expectation is taken over the distribution of the 
generating system and the probing schedule. 


While the cost can be unbounded for some schedules, the cost of the optimal 
schedule is always bounded. To see that, consider a round-robin schedule, S, that 
probes each node every n steps. Clearly no item is undiscovered in this schedule 
for more than n steps, and the expected number of items generated in an interval 
of n steps is Thus, Q^{t) < which implies cost(iS,7r) < 

Therefore, without loss of generality we can restrict our discussion 
to bounded cost schedules. Also, note that when the sequence {E [(5‘^(t)] 
converges we have cost (iS,7r) = lim E [Q‘^(t)l (Cesaro Means [6]). 

One can equivalently define the cost of a schedule in terms of the expected 
time that an item is in the system until it is discovered. 


Lemma 1. Let ujf be the expected waiting time of an item generated at node i 
until node i is probed by schedule S. Then 

n 

COSt{S,TT) = 'n'jUjf ■ 
i=l 


Proof. Following the definition of the cost function we have 


cost (5, tt) = Ihn 7 y] y] E [Qf {t')] = 


t^OO t 




lim 

t—¥CO 




= E 


TTjwf, 


where the last eqaulity is obtained by applying Little’s Law 


□ 


Corollary 1. A schedule that minimizes the expected number of undiscovered 
items in the system simultaneously minimizes the expected time that an item is 
undiscovered. 


Corollary 2. For any schedule S, costiS^n) > X]r=i^*- 

Proof. As mentioned above, when we probe a node i at time t we discover only 
the items that have been generated by time t — 1. Therefore, wf > 1, and by 
Lemma [I] the proof is complete. □ 


Now, our main problem is defined as the following: 

Definition 5 (Schedule Optimization). Given a generating vector tt and a 
positive integer c, find a c-schedule with minimum cost. 


When the generating vector is not known a priori to the algorithm the goal 
is to design a schedule that converges to an optimal one. For that we need the 
following definition: 


Definition 6 (Convergence). We say schedule S converges to schedule S', if 


for any generating vector tt, lim 


E[g5(t)]-E Q‘5'(t) 


= 0 . 


4 Results 

We start this section by, first, showing that the obvious approach of maximizing 
the expected number of detections at each step is far from optimal. We then 
prove a lower bound on the cost of any schedule, and provide deterministic and 
memoryless c-schedules that are within a factor of (3 + (c — l)/c) and (2 + 
(c — l)/c), respectively, from the optimal. Finally, we introduce an algorithm, 
Adaptive , which outputs a schedule A that converges to the optimal memoryless 
1-schedule when the generating vector tt is not known in advance. We also show 
that Adaptive can be used to obtain a c-schedule A" whose cost is within (2 -|- 
(c — l)/c) factor of any optimal c-schedule. 

Throughout this section, by rf (t) we mean the number of steps from the last 
time that node i was probed until time t, while executing schedule 5; if i has 









not been probed so far, we let rf^t) = t. Using the definition, it is easy to see 
that 

E[Qf(<)] =7r,E[Tf(t)], (1) 

when the expectations are over the randomness of both S and tt. Therefore, if 
the expectation is over only the randomness of tt we have 

E[Qf(t)] =7r,rf(t). (2) 


4.1 On Maximizing Immediate Gain 


Let iS be a 1-schedule that at each step, probes the node with the maximum 
expected number of undetected items. By ([S]), the expected number of undetected 
items at node i and at time t is TTiTf{t), and thus, S(t) = argmaxi iTirf (t). 

Now, suppose TTi = 2“®, for 1 < i < n. Since the probability that node 1 has 
an undetected item in each step is at least 1/2, node i is probed no more than 
once in each 2*“^ steps. Thus, the expected number of time steps that an item at 
node i will stay undetected is at least 2 t^(l + - ■ .-1-2*“^) = ^ > 2*“^. Using 

Lemmadl the cost of this schedule is at least = fl{n). 

Now, consider an alternative schedule that probes node i in each step with 
probability 2“*/^/Z, where Z = The expected number of steps 

between two probes of i is and the cost of this schedule is 






= 0 ( 1 ). 


Thus, optimizing immediate gain is not optimal in this problem. 


4.2 Lower Bound on Optimal Cost 

In this section we provide a lower bound on the optimal cost, i.e., the cost of an 
optimal schedule. 


Theorem 1. For any c-schedule O with finite cost we have 

/ ^ \ 2 ' 

1 


cost{0, tt) > max 


Proof. First, by Corollary O cost (O, tt) > show cost (O, tt) > 

^ (Sr=i V^) ■ positive integer t > 0, and suppose during the time in¬ 

terval [0,t], O probes node i at steps ti,t 2 , ■ ■ ■ Let to = 0 and t^+i = t. 
So, the sequence to,... ,trn+i partition the interval [0,t] into Ui + 1 intervals 




h{j) = [tj + for 0 < j < rii, and the length of Ii{j) is £i{j) = tj+i - tj. 

Applying the Cauchy-Schwartz inequality we have: 


n-i rii / Tii \ 

j=0 j=0 \j=o J 

j=0 \j=0 / 


e 


Ui + l 


-fi-—V 

Ui \ rii + l J 


For t' G Qf{t') is a Poisson random variable with parameter TTi{t' — tj). 

Therefore, 


^E[Qf(t')]=^ ^ E[Qf(0] =7r.^(l+ (j)) 

i^o j^o 




j=0 


+ 1) ^ T 




j=0 


2 Tli 




By summing over all nodes and averaging over i, we have 

n t 

' -¥. > V 

t 2 Tli 


1 1 f2 


i=l i' = l 


1 - 




rii + l 
1 

Tli + 1 


i=l 




\i=l 

2 


\i=l 


n* + 1 


1 


Ui + l 


(3) 


where in the second line we use the fact that if the schedule executed c probes 
in each step then ^ — o, and the third line is obtained by applying the 

Cauchy-Schwartz inequality. 

It remains to show that for any schedule with finite cost, and any i such that 
TTi > 0, lim rij = oo. For sake of contradiction assume that there is a time s such 

t—^OO 

that the node i is never probed by O at time t > s. So, E [Qf (t)] = n{t — s) and 
we have cost ( 0 , 7 r) > 7 X]t'=s [Q?] (^) = ^ (t-s)(t-s-i) converges to 

00 as t —>• 00 , which is a contradiction. Hence, for all i, lim rii = 00 , and using 

_ t—^OO 

(j4.2|) we obtain 


cost(0,7r) > lim — 
t —^00 2c 




which completes the proof. 


□ 















4.3 Deterministic (3 + (c — l)/c)-Approximation Schedule 

We construct a deterministic 1-schedule in which each node i is probed approx- 
imately every nt = — ^ steps, and using that, present our (3 -I- (c — l)/c)- 
approximation schedule. For each i let be a nonnegative integer such that 
2’’* > rii > 2’’*“^, and let p = maxi 

Lemma 2. There is a 2P-cyclic 1-schedule T) such that node i is probed exactly 
every 2’’* steps. 

Proof. Without loss of generality assume otherwise we can add 

auxiliary nodes to complete the sum to 1, with the powers (vi’s) associated with 
the auxiliary nodes all bounded by p. 

We prove the lemma by induction on p. If p = 0, then there is only one node, 
and the schedule is 1-cyclic. Now, assume the statement holds for all p' < p. 
Since the smallest frequency is 2“^, and the sum of the frequencies is 1, there 
must be two nodes, v and u, with same frequency 2~p. Join the two nodes to 
a new node w with frequency 2“^+^. Repeat this process for all nodes with 
frequency 2~p. We are left with a collection of nodes all with frequencies > 2~p. 
By the inductive hypothesis there is a (2^“^)-cyclic schedule V such that each 
node i is probed exactly each 2’'* steps. In particular a node w that replaced u 
and V is probed exactly each 2~p^^ steps. 

Now, we create an 2^-schedule, V, whose cycle is obtained by repeating the 
cycle of T)' two times. For each probe to w that replaced a pair u, v, in the first 
cycle we probe u and in the second cycle we probe v. Thus, u and v are probed 
exactly every 2^ steps, and the new schedule does not change the frequency of 
probing nodes with frequency larger than 2~p. □ 

Theorem 2. The cost of the deterministic 1-schedule T) is no more than 3 times 
of the optimal cost. 

Proof. By Lemma[2]each node i is probed exactly every 2’’* steps. Using 2’’’“^ < 
have 2’’* -|- 1 < ^ _|_ i and therefore 

\/T^i — V'n’i ’ 


■to 7 1 ^ [<3?(0] = irJW £ «[0?«')] = f- £ -.r = " 


< 


TT. 


+1 


2’'* 2 
TTi 


i=i 


Thus by Theorem [I] we have 


Th i Th ( Th \ IT 

cost(2?,7r) =^hin 

i=i i'=i i=i y j=i j i=i 
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i=i 


TT 







where cost (0,7r) is the optimal cost. 


□ 


Using the previous deterministic 1-schedule, the following corollary provides 
a c-schedule whose cost is within (3 -I- (c — l)/c) factor of the optimal cost. 


Corollary 3. There is a deterministic c-schedule whose cost is at most 
(3 -f (c — l)/c) times of the optimal cost. 


Proof. Consider the execution of the deterministic 1-schedule 2? constructed 
in Theorem [5] on generating vector -tt. Let 'D'^ be a deterministic c-schedule 
obtained by grouping c consecutive probes of V into one step. Suppose O is an 
optimal c-schedule. Applying equation ([TJ, 


cost tt) = ^ TT.wf ° ^cwf ° ^ -f c - 1) 


=cos.p,x)+y:u^<3(f: 


(c- l)7ri 




2=1 
\ 2 




E 


(C- l)lTi 


^2 = 1 


< 3 


c — 1 


E 

i=l 


cost (O, tt) 


where the first inequality holds because some items could be detected in less 
than c steps in the 1-schedule but are counted in one full step of the c-schedule. 
The last inequality is obtained by applying Theorem [T] □ 


4.4 On Optimal Memoryless Schedule 

Here, we consider memory less schedules, and show that the memory less 1-schedule 
with minimum cost can be easily computed. We call a memoryless schedule 
with minimum cost among memoryless schedules, an optimal memoryless sched¬ 
ule. We also provide an upper bound on the minimum cost of a memory less 
c-schedule. 


Theorem 3. LetTZ = (pi,... ,p„) be a memoryless Tschedule. Then cost{TZ,Tr) > 


V^) ’ equality holds if and only if pi = 




, for all i. 


Proof. Since probing each node i is a geometric distribution with parameter 
Pi, the expected time until an item generated in node i is discovered, is = 
l/pi. Therefore, by Lemma [U we have cost(7^, tt) = P* ~ 

argmin^^p cost (7?., tt), using the Lagrange multipliers: 


d 

dpj 




\i=l 


Pt 


i=\ 


= 0 


Pi « 







Therefore, cost (JZ,Tr) is minimized if Pi = and in this case the (mini¬ 

mized) cost will be 


n 

cost {TZ, tt) = 

i=l 




Corollary 4. The cost of the optimal memoryless 1-schedule is within a factor 
of 2 of the cost of any optimal 1-schedule. 


Proof. The cost of the schedule TZ in Theorem [3] is which is 

bounded by 2 • cost {O, tt) for an optimal 1-schedule O using Theorem [TJ □ 

Corollary 5. There is memoryless c-schedule, TZ‘^, whose cost is within a factor 
of {2 -\- (c — 1) /c) of any optimal c-schedule. 

Proof. Suppose is a memoryless c-schedule obtained by choosing c probes 
in each step, each chosen according to the optimal memoryless 1-schedule, TZ^ 
computed in Theorem[31 Using the same argument as in the proof of Corollary[3] 
we have 


cost {TZ°, tt) < 


1 

c 




cost (O, tt) , 


for an optimal c-schedule O. 


□ 


4.5 On Adaptive Algorithm for Memoryless Schedules 


Assume now that the scheduling algorithm starts with no information on the 
generating vector tt (or that the vector has changed). We design and analyze 
an adaptive algorithm, Adaptive, that outputs a schedule A convergent to the 
optimal memoryless algorithm TZ (see Section WM by gradually learning the 
vector TT by observing the system. To simplify the presentation we present and 
analyze a 1-schedule algorithm. The results easily scale up to any integer c > 

1, where the adaptive algorithm outputs a c-schedule convergent to 72.° (as in 
Section HU). 

Each iteration of the algorithm Adaptive starts with an estimate tt = (tti, ..., 7r„ 
of the unknown generating vector tt = (tti, ..., 7r„). Based on this estimate the 
algorithm chooses to probe node i with probability pi {t) = _ (which is 

the optimal memoryless schedule if Ttt was the correct estimate). If nodes i is 






probed at time t, the estimate of is updated to Wig ■(— ^ where Cig is 

the total number of new items discovered in that node since time 0. 

Algorithm 1: Adaptive 


Outputs: A(t), for t = 1, 2,.... 

begin 

(ci,...,c„) ^ (0, 

(ifi,... ,7f„) ^ 

for t = 1, 2,... do 


for i G {1,... 

Pi{t) 


, n} do 

y/Wj 




A{t) ^p{t)- 
output A{t); 

c! •<— number of new items caught at io; 

Cig i— Cig + c^; 

max(l,Cin) 

< - 


We denote the output of Adaptive schedule by A and the optimal memo¬ 
ryless 1-schedule by 7^ = p* = (p*,... ,Pn)', see Section l44l Our main result of 
this section is the following theorem. 

Theorem 4. The schedule A converges toTZ, and thus, cost(A, tt) = costiTZ,^). 

To prove Theorem |4] we need the following lemmas. 

Lemma 3. For any time t and i G [n] we have Pi{t) > 

Proof. It is easy to see that Pi{t) will reach its lowest value at time t only if for 
j ^ i we have tTj = 1 and rri = (which requires i to be probed at time t—1). 

Therefore, Pt{t) > = i+(„-l)Vt-i - irtr’ ^ 

Define S{t) = 4nexp ^ and let Nq be the smallest integer t such that 

exp < 2 exp ^*6 ■ Note that one can choose S{t) = An exp — 

for any e G (0,1/2), and for convenience we chose e = 1/6. 

Lemma 4. For any time t > Nq, with probability > 1 — S{t)/2, all the nodes 
are probed during the time interval [t/2,t) . 

Proof. By Lemma [31 the probability of not probing i during the time interval 
[t/2,t) is at most 


t-i 

t'=t/2 



^V/2 

ny/t) 


_ t 

< e < 2 exp 



2n 


A union bound over all the nodes completes the proof. 


□ 


















Lemma 5. Suppose node i is probed at a time t' > i/2. Then, 


Pr 


m') 



2n 


Proof. We estimate tt from t' > 112 steps, each with tt^ expected number of new 
items. Applying a Chernoff bound m for the sum of t' independent random 
variables with either Bernulli or Poisson distribution we have 


Pr 


\TTi{t') - TTil > t ^TTi 


< 2 exp — - 


t 3 7r, t' 


< 2 exp — - 


t 3 7r*i 


2n 


Note that by union bound, Lemma[5]holds, with probability at least l—5{t)/2, 
for all the nodes that are probed after i/2. 


Lemma 6. Suppose t > Nq. With probability at least 1 — d{t) we have for all 
i G [n], 


1 - 


1 


il/3 + 1 


p:< 


1 - i-V3 


l + i-l/3 


■n* < 1 + 


Proof Applying LemmalU Lemma[5]and a union bound, with probability 1—5(i) 
all the nodes are probed during the time [i/2,i) and |7ri(i) — TTij < for 

*i(t) 


all i G [n]. Since Pi(t) = 


Ej V (*) 


we obtain 


y., GiTTFvyy V1 


l-i-i/3 

-I- y:-l /3 V . . AF 7 “ 


l-i-1/3 


■P* > (1- 




where the last inequality uses the Taylor series of i/l + x. The upper bound is 
obtained by a similar argument. □ 


Corollary 6. The variation distance between the distributionp(t) = (pi(i), ■. ■ ,Pn{t)) 
used by algorithm Adaptive at time t > Nq, and the distributionp* = {p*,... ,p*) 
used by the optimal memoryless algorithm satisfy 

1 ” 

II P{i)-P* 11= -j^'^\Pi{t) - P*\ < + S{t) 0. 

i—\ 

Finally, we present our proof for Theorem |4l 


Proof of Theorem Recall that we defined rf (i) as the number of steps from 
the last time that node i was probed until time i in an execution of an schedule 
5, and E [Qf] = tt^E [rf (i)]. 


























Let F{t) indicate the event that the inequalities in Lemma [5] are held for 
Vt' € [t/2,t). Therefore, Pr[F(t)] < 1 — by applying union bound over 

all t' G [t/2,t), and using the fact that S{t') < 6(t/2). Therefore, 


|E [Q^it)] - E [Q^(f)] I 




< ^ TT, |E [T^(t)] - E [rf (f)] 
2=1 


< ^ TT, |E [rf'it)] - E [T^(f) I F{t)] 
2=1 


+ ^TTi |E I F{t)] - E [rf (t)] 

i=l 


where we used the triangle inequality for both inequalities. So, it suffices to show 
that for every i, 




ffin |E [r^(t)] - E [T^(t) | F{t)] \ = ffin |E [r^(t) | F(f)] - E [rj 

Obviously, < t. Now by letting t > 2No we have, 

E [r^(t)] = Pr [F(t)] E [T^{t) \ F{t)] + Pr [^F{t)] E [T^{t) \ -^F{t)] 


= 0 . 


< E [r^it) I F{t)] + = E [rf{t) \ F{t)] + ^ 


We also get 


E [rfit)] > (^1 - E [r^it) \ F{t)] 


(4) 


(5) 


= E [r^it) I F(t)] - ^^E [rf{t) \ F(t)] > E [r^(t) | X] - 


p/3 


Note that lim = lini 4ne e 'in = 0, and thus by (|4]) and ([5]) we have 


ffi^E [r^(f)] - E [r^(t) | Fit)] = 0 ^ ^Ihn |E [r^(t)] - E [r^(z) | Fit)] \ = 0. 

( 6 ) 


Now, we show that ^lim |E [r^it) \ F(t)] — E [rj^it)] | = 0. So here, we as¬ 
sume Fit) holds. So for every i G [n], node i is probed in [t/2,t), and for all 
t' G [f/2, t) we have 

(i) Prit') > (l - PttI+t) > (l - (t/2)V3+i ) pI So, 


E [r^(t) I < 1- 


(t/2)i/3-biy p* 


l = (i + it/2)-^/A\. 

Pi y 2 p* 















(ii) p,{t') < (l + P* < {^+ (t/2)i/3-i ) p*- Hence, 

E [r^m I no] > (i + ^ = (i - (02)--«) y 

Obviously E [rj^(t)] = since probing node i by TZ can be viewed as a geo¬ 
metric distribution with parameter p*, and since (5(t) —>■ 0 as t —>■ oo we have 

E [ry{t)] = ^ = hm (l - (t/2)-i/3) 1 < lim E [r^(t) | F{t)] 

< hm (l + (t/2)-H3) 1 = 1 = E [rf (t)] . 

t->-oo V / P* P* 

Therefore, 

|E [r^(t) I - E [rf (t)] | = 0. (7) 

Thus, by dH) and ([7]) we have lim |E \Q-^{t) \ — E | = 0, and A converges 

t^OO L J L J 

to S, and since lim E [Q‘®(t)l = implies that lim E r(5^(^)] = 

\/^)^ = cost {A, n) (by Cesaro Mean [B]). □ 

Note that one can obtain an adaptive schedule A‘^ by choosing c probes 
in each step, at each round of Adaptive, and using similar argument as in 
Section (and similar to Corollary [3]) , it is easy to see that A'^ converges to 

Finally, if tt changes, the Adaptive algorithm converges to the new optimal 
memoryless algorithm, as the change in the rate of generating new items is 
observed by Adaptive. 
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